Ideas and Challenges for using Synthetic Data to Augment Clinical Data

The emergence of generative AI and virtual patients

• We are starting to see more examples of using generative models to create clinical data

  • Generative models for imaging data –

    See Sizikova & CDRH colleagues (2024)¹ (GANs, diffusion models, deconvolutional models, VAEs, etc.)

  • AnimalGAN from NCTR (Chen et al. 2023²) –

    generation of synthetic clinical pathology measurements to assess toxicology of untested chemicals on animals

  • Digital Twins –

    Unlearn.AI’s PROCOVA (prognostic covariate adjustment model)³

Some Past Bayesian Methods for Generating Synthetic Data

Challenges with synthetic data

Bayesian methods for synthetic data generation

• Challenge 1: Can naturally incorporate prior information into the generative model (e.g., as an informative prior distribution on the model weights, as conditional information for the model). This may help generate data similar to real data.

• Challenge 2: Can incorporate uncertainty into the data generation process by using prior distributions on model parameters (leading to posterior distns on parameters).

  • Can generate diverse instances by sampling different parent parameters from the posterior distribution, then sampling instances given the parameters.

  • Traditional synthetic data generation methods may rely on point estimates of parameters and thus not fully capture the underlying data distribution.

Challenge 3: How do we borrow strength from synthetic data to estimate product performance on real patients?

• Bayesian hierarchical models (BHMs) are natural frameworks for combining data across sources

  • When estimating a parameter associated with the real data, a hierarchical model could borrow strength from synthetic data by assuming the parameters for the synthetic and real data are exchangeable, i.e., iid “draws” from same super-population.
  • Borrowing may improve the precision of the estimated parameter for the real data as the variation in parameters between synthetic and real data decreases.
    

Evaluating comparability of synthetic data to real source data

Evaluating comparability of synthetic data to source

• Traditional metrics to assess comparability
  • Qualitative: Compare latent representations of real versus synthetic data (e.g., PCA, tSNE, UMAP) to ensure that the generated datasets capture properties of the source data distribution.
  • Quantitative: Compute cosine similarity or KS-test per variable across datasets
  • Quantitative: Train a classifier to predict real versus fake samples
• Exchangeability - more a concept rather than a measure
  • Some borrowing methods assume exchangeability (e.g., HMs)
  • Regulatory Considerations – often rely on subject-matter experts, perhaps with some down-weighting of prior data

Example: Bayesian GAN to generate synthetic data

• GANs are deep neural network architectures consisting of a generator and a discriminator. They are set up to compete against each other (hence the term adversarial). Given some input data
  • the generator G tries to create truly new samples from the input data distribution, passing them on to the discriminator
  • the discriminator D receives both real and generated ata and tries to determine if what it gets was real or fake
  • G tries to fool D (make fake look real), who in turn tries to improve on distinguishing real vs. fake
• By placing prior distributions on the parameters of the generator and discriminator, the Bayesian GAN approximates a posterior distribution on the parameters and then generates synthetic data from the approximate posterior predictive distribution.

Bayesian GAN vs. traditional GAN

Compare generated data to source (validation) data

• Compare learned representations across synthetic and real datasets.

• Comparison of first 2 PCs of each of 10 synthetic datasets with validation dataset

• Compare histograms of validation data and (all) generated data across all 39 variables

• Nonetheless: Despite some issues with comparability of synthetic data with real validation data, we will forge ahead anyway…

Combine real and synthetic data using a BHM

Description of Bayesian Hierarchical Model

Bayesian hierarchical logistic regression model, where the synthetic datasets (\(j=1,...,10\)) serve as prior studies for the “real” study (\(j=test\)).

\[y_{ij} \sim Bern(p_{ij}) \hspace{5mm} i=1,...,n_j = 100; \hspace{3mm} j=1,...,10,\text{test}\] \[p_{ij} = (1+\operatorname{exp}(\alpha_j +\mathbf{x}_{ij}^{T}\mathbf{\beta}_j))^{-1} \]

\[\mathbf{\beta}_j \sim N(\mathbf{0},\mathbf{\Sigma}_{\beta}), \hspace{3mm} \alpha_j \sim N(0, 10) \hspace{3mm} j=1,...,10,\text{test}\] LKJ prior on the correlation matrix (then transform back to covariance matrix \(\mathbf{\Sigma_{\beta}}\))

\[Cor(\mathbf{\beta}) = \mathbf{R}_{\mathbf{\beta}} = L_\Omega^T L_\Omega \sim LKJ\_Corr(\eta = 0.8) \propto \operatorname{det}(\mathbf{R}_{\mathbf{\beta}})^{\eta - 1}\]

We want to make inference on \(\mathbf{\beta}_{test}\).

Comments on application of BHM to the example

• The synthetic datasets from the BayesGAN may have been too diverse compared to the test dataset resulting in minimal borrowing across coefficient vectors.

• A model with clusters of exchangeability may be more appropriate if some synthetic datasets are more similar to the current study than others. A Dirichlet process mixture model or LEAP model (Alt et al., 2024⁹) could flexibly model alternatives to full exchangeability.

• One could quantify how much was borrowed from synthetic datasets using prior effective sample size (PESS). PESS represents the amount of information contributed by the prior (synthetic datasets).

• Several proposals for computing PESS. Approximation may be necessary for more complicated models. Reimherr et al. (2021)¹⁰ provide an approximation for a “multivariate” PESS.

Concluding remarks

• Bayesian versions of synthetic data generation are potentially fruitful topics for future research.

  • Due the increasing availability of pre-trained models, these models might serve as priors to generate additional control subjects or even replace controls.

• BHMs are natural structures for combining information across synthetic datasets and real data. But, more flexible versions may be needed.

• Many newer generative models have natural levels of hierarchies that might be used (e.g., self-attention heads in multi-attention layers of transformers, generator networks in GANs)

• NumPyro (numpy backend for pyro with JAX) can be useful for fitting HMs with large amounts of simulated data.